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What GAO Found 


The Department of Veterans Affairs’ (VA) Veterans Heaith Administration (VHA) 
conducts nationai suicide prevention media outreach on various piatforms to 
raise awareness about VHA’s suicide prevention resources. The primary focus of 
this campaign since 2010 has been to raise awareness of the Veterans Crisis 
Line (VCL), VHA’s nationai hotiine estabiished in 2007 to provide support to 
veterans in emotionai crisis. GAO found that VHA’s suicide prevention media 
outreach activities deciined in recent years due to ieadership turnover and 
reorganization. For exampie, the amount of suicide prevention content 
deveioped by VHA’s contractor for sociai media decreased in fiscai years 2017 
and the first 10 months of 2018 after increasing in each of the 4 prior years. 


Suicide Prevention Outreach Content for Social Media, since Fiscal Year 2013 
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VHA officials reported not having leadership available for a period of time to 
make decisions about the suicide prevention media outreach campaign. GAO 
found that VHA did not assign key leadership responsibilities or establish clear 
lines of reporting, and as a result, its ability to oversee the outreach campaign 
was hindered. Consequently, VHA may not be maximizing its reach with suicide 
prevention media content to veterans, especially those who are at-risk. 

VHA evaluates the effectiveness of its suicide prevention media outreach 
campaign by collecting data on metrics, such as the number of people that visit 
the VCL website. However, VHA has not established targets for the majority of 
these metrics. Officials said they have not established targets because, apart 
from one industry-wide target they use, they lack meaningful targets for 
evaluating the campaign. However, VHA could use information about how its 
metrics performed in the past to develop reasonable and meaningful targets for 
future performance. Without established targets for its metrics, VHA is missing 
an opportunity to better evaluate the effectiveness of its suicide prevention media 
outreach campaign. 
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REACH VET 

VA 

VCL 
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Centers for Disease Control and Prevention 
Office of Mental Health and Suicide Prevention 
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Health - Veterans Enhanced Treatment 
Department of Veterans Affairs 
Veterans Crisis Line 
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om 

441 G St. N.W. 
Washington, DC 20548 


U.S. GOVERNMENT ACCOUNTABILITY OFFICE 


November 15, 2018 

The Honorable Tim Walz 
Ranking Member 
Committee on Veterans’ Affairs 
House of Representatives 

Dear Mr. Walz: 

Suicide is a public health problem facing all populations, particularly 
veterans. After military service, veterans suffer a disproportionately higher 
rate of suicide than the civilian population. According to the Department of 
Veterans Affairs (VA), veterans accounted for 14.3 percent of all deaths 
by suicide among United States adults in 2015, despite constituting only 
8.3 percent of the adult population. VA reported in June 2018 that an 
average of 20 veterans died by suicide each day in 2015, and that an 
average of 6 of these 20 veterans (30 percent) were recent users of VA’s 
Veterans Health Administration’s (VHA) health care services.'' Suicide is 
not unique to veterans; the Centers for Disease Control and Prevention 
(CDC) reports that suicide was the tenth leading cause of death in the 
United States in 2015.^ However, according to VA, the rate of suicide is 
higher for veterans than non-veterans (see figure 1). 


^See Department of Veterans Affairs, VA National Suicide Data Report (Washington, 
D.C.: June 2018). VA’s June 2018 report is based on 2015 data, the most recent year of 
data available. Recent users of VHA health care services is defined as those who 
received VHA care in 2015 or 2014. 

^See Centers for Disease Control and Prevention, Preventing Suicide: A Technical 
Package of Policy, Programs, and Practices (Atlanta, Ga.: 2017). 
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Figure 1: Veteran Suicide Rates Compared to Non-Veterans, 2015 
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An average of 117 adults die by 
suicide each day. 



of them are veterans. 



6 of those 20 veterans were recent users of 
Veterans Health Administration (VHA) services. 




The rate of suicide is 
two times higher for 
veterans than it is for 
non-veterans. 


Source: Department of Veterans Affairs (VA). | GAO-19-66 

Note: See Department of Veterans Affairs, VA National Suicide Data Report (Washington, D.C.: June 
2018). The report inciudes data from 2015, the most recent year of compiete data avaiiabie. in its 
agency comments, VA indicated that the rate of suicide is now one and a haif times higher for 
veterans than it is for non-veterans, based on its September 2018 report, which inciudes data through 
2016. See VA, VA National Suicide Data Report (Washington, D.C.: September 2018). 


VHA conducts national outreach to veterans and others in the community 
to raise awareness about suicide prevention and services available 
through VA, as one of several VHA initiatives to help reduce suicides in 
the veteran community. Since 2010, VHA has awarded multiple contracts 
to the same vendor to develop content for suicide prevention media 
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outreach.^ Questions have been raised about the effectiveness of VHA’s 
collective suicide prevention media outreach activities, also referred to as 
its suicide prevention media outreach campaign. You asked us to review 
VHA’s suicide prevention media outreach activities and related oversight. 
In this report, we examine: 

1. the extent to which VHA conducts activities for its suicide prevention 
media outreach campaign; and 

2. the extent to which VHA evaluates the effectiveness of its suicide 
prevention media outreach campaign. 

To examine the extent to which VHA conducts activities for its suicide 
prevention media outreach campaign, we reviewed VHA’s contract to 
develop suicide prevention outreach content from fiscal years 2013 
through 2016 and its contract to develop suicide prevention and mental 
health outreach content in fiscal years 2017 and 2018."^ VHA’s first such 
contract began in 2010; however, the Federal Acquisition Regulation only 
requires federal agencies to maintain contract documentation for a period 
of 6 years after the last payment. Therefore, we did not include the initial 
contract or associated outreach in our review due to the lack of available 
documentation. We also requested and reviewed data on the amount and 
type of outreach content that VHA’s contractor developed from fiscal year 
2013 through fiscal year 2018.® We interviewed VHA officials responsible 
for implementation and oversight of the contract about the data. We also 
obtained written responses from the contractor about the steps it took to 
compile and ensure the accuracy of the data. On the basis of these 
actions, we found these data to be sufficiently reliable for the purpose of 
our reporting objective. 


^Outreach “content” refers to materials for communicating with the public, such as flyers, 
and billboards, and social media posts. VHA’s suicide prevention content typically includes 
images and information about its suicide prevention services and resources. 

^Both contracts are with the same vendor, which we will refer to hereafter as VHA’s 
“contractor.” 

®ln order to expedite this data request, VHA enlisted the assistance of the contractor to 
provide the data in the format and time frames required for this review. We requested the 
data on a subset of the types of content the contractor develops in order to focus on those 
types of media outreach most relevant to our work. For example, we did not focus on 
promotional items, such as magnets and keychains, and data on website maintenance. 

For fiscal year 2018, we obtained data through July 2018, the first 10 months of the fiscal 
year. 
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We also reviewed documentation and interviewed VHA officials who 
worked on the campaign about VHA’s suicide prevention media outreach 
campaign, outreach activities, and VHA leadership officials’ involvement 
in the campaign, and plans moving forward.® We compared VHA’s 
outreach campaign and related activities with federal internal control 
standards related to control environment and monitoring.^ We also 
interviewed relevant other VHA and VA officials who played a role in the 
suicide prevention media outreach campaign, including officials in VA’s 
Office of Acquisitions, Logistics, and Construction and VHA’s Office of 
Communications. 

To examine the extent to which VHA evaluates the effectiveness of its 
suicide prevention media outreach campaign, we reviewed 
documentation of VHA’s evaluation efforts, including monthly monitoring 
reports, to identify any performance metrics VHA monitors and any 
targets established for those metrics. We also reviewed the 2016 contract 
to identify any requirements for the contractor to monitor and report on 
the effectiveness of the campaign. Additionally, we interviewed VHA 
officials responsible for monitoring and evaluating the campaign and 
reviewed supporting documentation. Specifically, we asked officials about 
the extent to which they had evaluated the campaign’s effectiveness 
since fiscal year 2013, their evaluation processes and tools, and any 
plans for completing future evaluations. We then compared VHA’s 
evaluation efforts to criteria we developed in previous bodies of work for 
designing program evaluations, as well as to federal internal control 
standards for monitoring.® 


^Officials that worked on the campaign include the contracting officer’s representative (the 
primary VHA official responsible for oversight of the contract) and other officials within 
VHA that worked directly on the campaign. 

^GAO, Standards for Internal Control In the Federal Government, GAO-14-704G 
(Washington, D.C.: Sept. 10, 2014) and Standards for Internal Control In the Federal 
Government, GAO/AIMD-00-21.3.1 (Washington, D.C.: Nov. 1, 1999). Internal control is a 
process effected by an entity’s management, oversight body, and other personnel that 
provides reasonable assurance that the objectives of an entity will be achieved. 

®GAO, Tax Administration: IRS Needs to Further Refine Its Tax Filing Season 
Performance Measures, GAO-03-143 (Washington, D.C.: Nov. 22, 2002). Our prior work 
states that clearly articulating a program’s strategy and performance expectations aids the 
selection of appropriate performance measures and data sources. For example, a well- 
articulated program logic model can help evaluate agency efforts. See also GAO, 
Designing Evaluations: 2012 Revision, GAO-12-208G (Washington, D.C.: Jan. 31,2012), 
GAO-14-704G, and GAO/AIMD-00-21.3.1 . 
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We conducted this performance audit from December 2017 to November 
2018 in accordance with generaiiy accepted government auditing 
standards. Those standards require that we pian and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonabie basis for 
our findings and conciusions based on our audit objectives. We beiieve 
that the evidence obtained provides a reasonabie basis for our findings 
and conciusions based on our audit objectives. 


Background 


VA Suicid© Pr6V6ntion VA has undertaken a number of initiatives to heip prevent veteran suicide, 

inciuding identifying suicide prevention as VA’s highest ciinicai priority in 
its strategic pian forfiscai years 2018 through 2024 (see fig. 2).® 


®See Department of Veterans Affairs, Department of Veterans Affairs Fiscal Years 2018- 
2024 Strategic Plan (Washington, D.C.: Feb. 12, 2018). 
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Figure 2: Top Five Priorities in the Department of Veterans Affairs’ (VA) Strategic Pian for Fiscai Years 2018 - 2024 
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GREATER CHOICE 

VA is committed to ensure veterans can 
make decisions that work best for them 
and their famiiies. 



MODERNIZE SYSTEMS 


o 


Veterans and VA empioyees need 
systems and technoiogy that 
enabie them to deiiver the 
high-quaiity care and services 
veterans deserve. 
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IMPROVE TIMELINESS 

Veterans must receive the benefits, 
care, and services they need in 
a timeiy manner, no matter where 
they are. 


o 






SUICIDE PREVENTION 

Suicide prevention is VA’s highest 
clinical priority. Suicide is a 
national health crisis; it 
requires all of government, 
along with pubiic-private 
partnerships to address. 



FOCUS RESOURCES 

Veterans and taxpayers deserve to 
know VA resources are spent on the 
care and services veterans need most. 


VETERANS CRISIS LINE 

1-800-273-8255 
Press 1 






Source: VA (information); GAO (illustration). | GAO-19-66 


VA uses CDC’s research on risk factors and prevention techniques to 
inform its approach to suicide prevention in the veteran community. There 
is no single determining cause for suicide; instead, suicide occurs in 
response to biological, psychological, interpersonal, environmental, and 
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societal influences, according to the CDC. Specifically, suicide is 
associated with risk factors that exist at the individual level (such as a 
history of mental illness or substance abuse, or stressful life events, such 
as divorce or the death of a loved one), community level (such as barriers 
to health care), or societal level (such as the way suicide is portrayed in 
the media and stigma associated with seeking help for mental illness). 
According to VA, veterans may possess risk factors related to their 
military service, such as a service-related injury or a difficult transition to 
civilian life. CDC reports that protective factors—influences that help 
protect against the risk for suicide—include effective coping and problem¬ 
solving skills, strong and supportive relationships with friends and family, 
availability of health care, and connectedness to social institutions such 
as school and community. 

VA’s 2018 National Strategy for Suicide Prevention identifies four focus 
areas: (1) healthy and empowered veterans, families, and communities; 
(2) clinical and community preventative services; (3) treatment and 
support services; and (4) surveillance, research, and evaluation.'"’ 
Collectively, these four areas encompass 14 goals for preventing veteran 
suicide, one of which is implementing communication designed to prevent 
veteran suicide by changing knowledge, attitude, and behaviors. 

VHA’s suicide prevention media outreach campaign is just one of its 
initiatives intended to reduce veteran suicide. For example, in 2007, VHA 
established the Veteran’s Crisis Line (VCL), a national toll-free hotline 
that supports veterans in emotional crisis.Veterans, as well as their 
family and friends, can access the VCL by calling a national toll-free 
number—1-800-273-8255—and pressing “1” to be connected with a VCL 
responder, regardless of whether these veterans receive health care 
through VHA. VHA added the option to communicate with VCL 
responders via online chat in 2009, followed by text messaging in 2011. 
Another VHA suicide prevention initiative is the Recovery Engagement 


1 n 

Department of Veterans Affairs, National Strategy for Preventing Veteran Suicide 2018- 
2028 (Washington, D.C.: June 29, 2018). 

’’We have previously reported on the VCL and made recommendations for improving the 
timeliness and quality of VA’s VCL responses. VA concurred with our recommendations 
and in July 2016, developed and implemented procedures to regularly test the VCL 
system. In addition, in July 2017, VA updated its VCL quality assurance plan to document 
measurable targets and time frames for key performance indicators needed to assess the 
VCL’s performance. See GAO, Veterans Crisis Line: Additional Testing, Monitoring, and 
Information Needed to Ensure Better Quality Service, GAO-16-373 (Washington, D.C.: 
May 26, 2016). 
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and Coordination for Health - Veterans Enhanced Treatment initiative, or 
REACH VET. Established in 2016, REACH VET uses predictive modeling 
to analyze existing data from veterans’ health records to identify veterans 
at increased risk for adverse outcomes, such as suicide, hospitalization, 
or illness. 


VHA’s Suicide Prevention 
Media Outreach 
Campaign 


Suicide prevention officials within VHA’s Office of Mental Health and 
Suicide Prevention (OMHSP) are responsible for implementing the 
suicide prevention media outreach campaign. Since 2010, VHA has used 
a contractor to develop suicide prevention media outreach content and 
monitor its effectiveness. In September 2016, VHA awarded a new 
contract to the same contractor to provide both suicide prevention and 
mental health media outreach. Under the 2016 contract, the suicide 
prevention and mental health outreach campaigns remain separate and 
are overseen by separate suicide prevention and mental health officials, 
both within OMHSP.VHA officials told us that beginning in fiscal year 
2019, VHA will separate the contract for suicide prevention and mental 
health media outreach. Specifically, VHA will utilize an existing agreement 
with a different contractor for suicide prevention media outreach while the 
existing contractor will continue to provide mental health media 
outreach. 


According to VHA, the purpose of its suicide prevention media outreach 
campaign is to raise awareness among veterans, their families and 
friends, and the general public about VHA resources that are available to 
veterans who may be at risk for suicide. The primary focus of the 


^^After being identified by REACH VET, veterans are contacted by VA health care 
providers to determine whether additional care or services are needed. 

^^VHA officials said OMHSP was established in May 2017. Before May 2017, the suicide 
prevention program and the mental health program were organizationally located under 
different offices within VHA. In addition to these two programs, OMHSP also includes the 
VOL program, among others. 

14 

Specifically, VHA said it will utilize an existing blanket purchase agreement through a 
GSA schedule contract. A blanket purchase agreement is a simplified method of filling 
anticipated repetitive needs for supplies or services that functions as a charge account, 
with terms and conditions agreed upon when the blanket purchase agreement is 
established. Federal Acquisition Regulation §§ 8.405-3, 13.303-1 (a). 
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outreach campaign since 2010 has been to raise awareness of the 
services available through the VCL.”'^ 

VHA’s suicide prevention media outreach falls into two main categories: 
unpaid and paid. Unpaid media outreach content is typically displayed on 
platforms owned by VA or VHA, or is disseminated by external 
organizations or individuals that share VHA suicide prevention content at 
no cost, as discussed below (see fig. 3).''® 

• Social media. VA and VHA each maintain national social media 
accounts on platforms such as Facebook, Twitter, and Instagram, 
and post content, including suicide prevention content developed 
by VHA’s contractor.''^ VHA also works with other federal 
agencies, non-governmental organizations, and individuals that 
post its suicide prevention content periodically. 

• Public service announcements (PSA). VHA’s contractor 
typically develops two PSAs per year, which various local and 
national media networks display at no cost to VHA. 

• Website. VHA’s contractor maintains the content displayed on the 
VCL website (veteranscrisisline.net), including much of the 
content it develops for other platforms, such as PSAs and social 
media content. Visitors to the website can both view the content 
on the website and share it on their own platforms. 


'®See GAO-16-373. 

^®VHA also conducts unpaid outreach in the form of attending conferences and seminars, 
as well as developing materials to hand out to veterans and the community at these and 
other events. However, this type of outreach was not the focus of our report because it is 
not a national, media-based outreach effort. 

^^VHA has a VCL Facebook page, but officials told us that they intentionally do not 
populate it with suicide prevention content because it would not be an appropriate 
resource for providing timely support to veterans in emotional crisis. 
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Figure 3: Examples of the Veterans Health Administration’s (VHA) Unpaid Suicide 
Prevention Media Outreach Content 
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Source: © J.R. Reingold & Associates, Inc. | GAO-19-66 

Note: Unpaid media refers to outreach content that is displayed on platforms owned by the 
Department of Veterans Affairs, or is disseminated by external organizations or individuals that share 
VHA suicide prevention content at no cost. 


Paid media outreach includes outreach on media platforms for which VHA 
must pay a fee to display its content (see fig. 4). Paid media falls into two 
categories, as discussed below. 

• Paid digital media. An example of paid digital media includes 
online keyword searches, in which VHA pays a search engine a 
fee for its website to appear as a top result in response to selected 
keywords, such as “veterans crisis line” or “veteran suicide.” Paid 
digital media also includes social media posts for which VHA pays 
a fee to display its content to a widespread audience, such as 
users with a military affiliation. 
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• Paid “out-of-home” media: “Out-of-home” refers to the locations 
where this type of content is typically displayed. Examples include 
billboards, bus and transit advertisements, and local and national 
radio commercials. 


Figure 4: Examples of the Veterans Health Administration’s Paid Suicide Prevention Media Outreach Content 



Example of an online keyword 
search on a search engine website. 



Example of a billboard advertisement. Example of a public transit advertisement. 


Source: © J.R. Reingold & Associates, Inc. | GAO-19-66 


VHA recognizes September as Suicide Prevention Month each year. 
During this month, VHA establishes a theme and increases its outreach 
activities, including a combination of both paid and unpaid media 
outreach. According to VHA, it typically incorporates additional outreach 
techniques during this month, such as enlisting the support of celebrities 
or hosting live chat sessions on social media platforms, including 
Facebook and Twitter. 
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VHA’s Suicide 
Prevention Media 
Outreach Activities 
Declined in Recent 
Years Due to 
Leadership Turnover 
and Reorganization 

VHA’s Suicide Prevention 
Media Outreach Activities 
Declined in Fiscal Years 
2017 and 2018 


VHA’s suicide prevention media outreach activities declined in fiscal 
years 2017 and 2018 compared to earlier years of the campaign. We 
identified declines in social media postings, PSAs, paid media, and 
suicide prevention month activities, as discussed below. 

• Social media. The amount of social media content developed by 
VHA’s contractor decreased in 2017 and 2018, after increasing in 
each of the prior four years. Specifically, VHA reported that its 
contractor developed 339 pieces of social media content in fiscal 
year 2016, compared with 159 in fiscal year 2017, and 47 during 
the first 10 months of fiscal year 2018 (see fig. 5.).''® 


18 

The data includes the amount of content that was developed, not the number of times it 
was posted or displayed. This is because the outreach content developed by VHA’s 
contractor is often disseminated on platforms that are not owned by VHA or its contractor. 
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Figure 5: Suicide Prevention Outreach Sociai Media Content Deveioped for the 
Veterans Heaith Administration, October 2012 through Juiy 2018 
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Source; Veterans Health Administration. | GAO-19-66 

Note: The table includes the amount of social media content developed, not the number of times it 
was posted on social media platforms or displayed in other locations. 

• PSAs. VHA’s contractor is required to develop two suicide 
prevention PSAs in each fiscal year. VHA officials said that the 
development of the two PSAs was delayed in fiscal year 2018. 
Specifically, as of August 2018, VHA reported that one PSA was 
completed, but had not yet aired, and another PSA was in 
development.''® As a result of this delay, VHA had not aired a 
suicide prevention PSA on television or radio in over a year; this is 
the first time there has been a gap of more than a month since 
June 2012. 


1Q 

VHA released a PSA on suicide prevention in October 2017. However, VHA officials 
said this PSA was not developed by the contractor, but through a partnership with an 
outside organization, and it was developed for online dissemination, as opposed to 
television or radio. 
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• Paid media. VHA had a total budget of $17.7 million for its suicide 
prevention and mental health media outreach for fiscal year 2018, 
of which $6.2 million was obligated for suicide prevention paid 
media. As of September 2018, VHA said it had spent $57,000 of 
its $6.2 million paid media budget.VHA officials estimated that 
they would spend a total of $1.5 million on suicide prevention paid 
media for fiscal year 2018 and indicated that the remaining funds 
would be de-obligated from the contract at the end of the fiscal 
year and not used for suicide prevention media outreach. VHA 
officials indicated that the reason they did not spend the remaining 
funds on suicide prevention paid media in fiscal year 2018 was 
that the approval of the paid media plan was delayed due to 
changes in leadership and organizational realignment of the 
suicide prevention program. As a result, VHA officials said they 
limited the paid media outreach in fiscal year 2018 to activities that 
were already in place, including 25 keyword search 
advertisements, and 20 billboards and 8 radio advertisements in 
selected cities across the United States. 

In prior fiscal years, VHA conducted a variety of digital and out-of¬ 
home suicide prevention paid media. For example, in fiscal year 
2015, with a suicide prevention paid media budget of more than 
$4 million, VHA reported that it ran 58 advertisements on Google, 
Bing, and Facebook, and ran 30 billboards, 180 bus 
advertisements, more than 19,000 radio advertisements, 252 print 
advertisements, and 39 movie theatre placements in selected 
cities across the United States.^'' VHA ran similar types of paid 
media in fiscal years 2013, 2014, and 2016 with variation in 
quantities based on the approved budget in each of these years. 

In fiscal year 2017, VHA had a budget of approximately $1.7 
million to spend on paid media for both the suicide prevention and 
mental health outreach campaigns. However, VHA spent less than 
10 percent of the funds (approximately $136,000) to run paid 
advertisements on Google and Bing for suicide prevention in fiscal 
year 2017; the remainder was spent on mental health outreach. 

• Suicide Prevention Month. VHA documentation indicated that 
Suicide Prevention Month 2017 was a limited effort. VHA officials 


prj 

VHA officials told us that because paid media is very effective, the amount of approved 
funds would be the biggest factor in determining the amount and type of paid media to 
conduct each year. 

^^In fiscal years 2014, 2015, and 2016, VHA paid movie theatres to show suicide 
prevention PSAs during the previews. 
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said that this was because they did not begin preparing early 
enough. In May 2018, VHA officials indicated that they were 
similarly behind schedule for planning Suicide Prevention Month 
2018, though they told us in August 2018 that they had caught up. 

VHA Leadership Turnover 
and Reorganization 
Resulted in the Decline in 
Suicide Prevention Media 
Outreach Activities 

VHA officials told us that the decrease in suicide prevention media 
outreach activities was due to leadership turnover and reorganization 
since 2017. For example, VHA officials said the National Director for 
Suicide Prevention position was vacant from July 2017 through April 

2018. VHA filled the role temporarily with a 6-month detail from another 
agency from October 2017 through March 2018 and then hired this 
individual as the permanent director on April 30, 2018. VHA officials that 
worked on the campaign told us they did not have leadership available to 
make decisions about the suicide prevention campaign during this time. 
For example, VHA officials said they did not have a kick-off meeting 
between VHA leadership and VHA’s contractor at the beginning of fiscal 
year 2018—a requirement of the contract—because there was no 
leadership available to participate in this meeting. 

The officials also reported that suicide prevention leadership was not 
available for weekly meetings to discuss suicide prevention outreach 
activities, even after the suicide prevention program obtained an acting 
director on detail from another agency. VHA staff said that at that time, 
they focused their suicide prevention media outreach efforts on areas that 
did not require leadership input, such as updating the VCL website. The 
absence of leadership available to provide direction and make decisions 
on the suicide prevention media outreach campaign is inconsistent with 
federal internal control standards for control environment, which require 
agencies to assign responsibilities to achieve its objectives.If a key role 
is vacant, management needs to determine by whom and how those 
responsibilities will be fulfilled in order to meet its objectives. 

Officials that worked on the campaign told us they shifted their focus 
away from the suicide prevention media outreach campaign toward the 
mental health outreach campaign due to reorganization of the offices 
responsible for suicide prevention activities in 2017. Specifically, under 

^^During kick-off meetings, VHA and its contractor typically discuss lessons learned from 
the prior year’s campaign and the approach for the coming year, such as target 
audiences, outreach platforms, paid media, and new initiatives. 

23gAO- 14-704G and GAO/AIMD-00-21.3.1. 
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the new organization, and in the absence of suicide prevention program 
leadership, the officials began reporting directly to mental health program 
leadership and became more focused on the mental health outreach 
aspects of the contract.Following the reorganization, officials that 
worked on the campaign did not have a clear line of reporting to the 
suicide prevention program. This is also inconsistent with federal internal 
control standards for control environment, which require agencies to 
establish an organizational structure and assign responsibilities, such as 
establishing lines of reporting necessary information to management.^® 

VHA officials told us that one of the highest priorities for the suicide 
prevention program since the beginning of fiscal year 2018 was to 
establish a national strategy for preventing veteran suicides. The national 
strategy, issued in June 2018, includes suicide prevention outreach as 
one of the strategy’s 14 goals.The national strategy also emphasizes 
VHA’s plans to shift to a public health approach to suicide prevention 
outreach. The public health approach focuses less on raising awareness 
of the VCL and more on reaching veterans before the point of crisis. 

VHA officials told us they have been trying to shift to a public health 
approach since 2016. Some of the campaign themes and messages have 
reflected this shift; for example, the “Be There” campaign theme that was 
adopted in fiscal year 2016—and has remained the theme since— 
emphasizes the message that everyone has a role in helping veterans in 
crisis feel less alone and connecting them to resources. However, VHA 
officials told us in May 2018 that they were just beginning to 
conceptualize what the suicide prevention outreach campaign should look 
like moving forward. Leadership officials also said that while they were 
developing the national strategy, they delegated the responsibility for 
implementing the suicide prevention outreach campaign to other officials 
working on the campaign. 


officials said that they had planned to appoint new staff to work specifically on the 
suicide prevention outreach campaign, but they had difficulty doing so until March 2018. 

2®GAO-14-704G and GAO/AIMD-00-21.3.1. 

^®VA, National Strategy for Preventing Veteran Suicide 2018-2028. 

^^Some VHA officials indicated that the shift away from raising awareness of the VCL was 
driven in part by reports that the VCL volume had exceeded its capacity for a period of 
time. However, VHA leadership officials, including the Director of the VCL, said that 
raising awareness of the VCL continues to be a priority. 
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The decline in VHA’s suicide prevention media outreach activities over 
the past 2 fiscal years is inconsistent with VA’s strategic goals, which 
identify suicide prevention as the agency’s top clinical priority for fiscal 
years 2018 through 2024.Further, VHA has continued to obligate 
millions of dollars to its suicide prevention media outreach campaign each 
year. Since fiscal year 2017, VHA has obligated $24.6 million to the 
contract for media outreach related to both suicide prevention and mental 
health.^® 

By not assigning key leadership responsibilities and clear lines of 
reporting, VHA’s ability to oversee the suicide prevention media outreach 
activities was hindered and these outreach activities decreased. As a 
result, VHA may not have exposed as many people in the community, 
such as veterans at risk for suicide, or their families and friends, to its 
suicide prevention outreach content. Additionally, without establishing 
responsibility and clear lines of reporting, VHA lacks assurance that it will 
have continuous oversight of its suicide prevention media outreach 
activities in the event of additional turnover and reorganization in the 
future, particularly as VHA begins implementing the suicide prevention 
media outreach campaign under its new agreement that begins in fiscal 
year 2019. 


^®VA, Department of Veterans Affairs Fiscal Years 2018- 2024 Strategic Plan. 

oq 

Some of the deliverables in the fiscal year 2017 contract apply only to one campaign, 
while other deliverables are divided evenly between the two campaigns. Between fiscal 
years 2013 and 2016, VHA obligated over $16.8 million to the contract specifically for 
suicide prevention media outreach. These amounts include the cost of paid and unpaid 
media outreach, as well as indirect costs. 
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VHA Monitors Metrics 
for Its Suicide 
Prevention Media 
Outreach Campaign, 
but Has Not 
Established Targets 
against Which to 
Evaluate the 
Campaign’s 
Effectiveness 


VHA Monitors Metrics for 
Its Suicide Prevention 
Media Outreach 
Campaign 


VHA works with its contractor to create and monitor metrics to help gauge 
the effectiveness of its suicide prevention media outreach campaign in 
raising awareness among veterans and others about VHA services, such 
as the VCL. The metrics primarily focus on the number of individuals who 
were exposed to or interacted with VHA’s suicide prevention content 
across various forms of outreach, including social media, PSAs, and 
websites.According to VHA, the metrics are intended to help VHA 
ensure that its media outreach activities achieve intended results, such as 
increasing awareness and use of the resources identified on the VCL 
website.^'' Examples of metrics monitored by VHA and its contractor 
include those related to (1) social media, such as the number of times a 
piece of outreach content is displayed on social media; (2) PSAs, such as 


on 

Specifically, for its suicide prevention media outreach, VHA tracks reach, impressions 
and engagement. For the purposes of this report, “reach” refers to the number of unique 
people who may have seen a piece of outreach content, while “impressions” typically 
estimate the total number of times a piece of outreach content was displayed to people. 
For example, if VHA posts two tweets and 100 people see both of them, the reach is 100 
while the impressions are 200. Meanwhile, “engagement” typically measures the number 
of times the audience interacted with a piece of outreach content, including click-through 
rates (the percentage of people exposed to an advertisement who click on it to access the 
advertised website) and social media “likes.” These metrics can help VHA gauge the 
extent to which its outreach is raising awareness of the VCL and other VHA services 
through its outreach. 

^VhA officials acknowledged that some metric outcomes can be affected by events or 
factors outside of VHA’s control. For example, certain public events can raise awareness 
of suicide risk and increase calls to the VCL or visits to the VCL website. 
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the total number of markets and television stations airing a PSA; and (3) 
the VCL website, such as the total traffic to the website, as well as the 
average amount of time spent on a page and average number of pages 
viewed per visit. 

VHA’s contractor is required to monitor the metrics and report results on a 
monthly basis. Specifically, the contractor provides monthly monitoring 
reports to VHA that summarize how outreach is performing, such as the 
number of visits to the VCL website that were driven from paid media 
sources. Officials noted these reports are key sources of information for 
VHA on the results of its outreach. VHA officials also told us they 
informally discuss certain metrics during weekly meetings with VHA’s 
contractor. In addition, VHA works with its contractor to conduct a more 
in-depth analysis of outreach efforts during and after Suicide Prevention 
Month each year. 


VHA Lacks Metric Targets 
to Evaluate the 
Effectiveness of Its Suicide 
Prevention Media 
Outreach Campaign 


VHA has not established targets for the majority of the metrics it uses to 
help gauge the effectiveness of its suicide prevention media outreach 
campaign. As a result, VHA does not have the information it needs to fully 
evaluate the campaign’s effectiveness in raising awareness of VHA’s 
suicide prevention resources among veterans, including the VCL.^^ For 
example, we found that VHA’s contractor’s monitoring reports—a 
summary of key metrics that VHA uses to routinely monitor information 
regarding the campaign—generally focused on outreach “highlights” and 
positive results. The reports did not set expectations based on past 
outreach or targets for new outreach, and lacked more comprehensive 
information on whether outreach performed against these expectations. 
For example: 


• A monitoring report from 2018 showed that during one month, 
there were 21,000 social media mentions of keywords specific to 
VA suicide prevention, such as “VCL” or “veteran suicide,” across 
social media platforms. These mentions earned 120 million 
impressions; however, there was no indication of the number of 


op 

While VHA has not established targets for the majority of its metrics, we found that VHA 
had established one specific target for its suicide prevention PSAs. Specifically, VHA 
officials told us their goal for each PSA is to rank in the top 10 percent of the Nielsen 
ratings; VHA’s contractor tracks these metrics and provides feedback to VHA through PSA 
monitoring reports. Nielsen ratings are a tool used to measure audience television 
viewership and are produced by The Nielsen Company (U.S.), LLC, a data analytics 
company. 
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keyword mentions or impressions that VHA expected based on its 
media outreach activities. In addition, the report did not indicate 
the proportion of mentions that VHA believed were specifically 
driven by its outreach activities, and there also was no indication 
of whether these mentions were positive or negative, or what 
actions to take based on this information.^^ 

• Another monitoring report from January 2017 showed that paid 
advertising drove 39 percent of overall website traffic during one 
month, while unpaid sources drove the remaining 61 percent. 
However, there was no information indicating the amounts of paid 
advertising that VHA conducted during this monitoring period, and 
whether this amount of website traffic from paid advertising met 
expectations. 

• VHA’s 2016 Suicide Prevention Month summary report showed 
that there were 194,536 visits to the VCL website, roughly an 8 
percent increase from the Suicide Prevention Month in 2015. 
However, the report did not indicate whether this increase from 
the prior year met expectations, or a different result was expected. 

VHA officials told us that they have not established targets for most of the 
suicide prevention media outreach campaign because they lack 
meaningful targets for the metrics to help evaluate the campaign. VHA 
officials said that the only target they have established is for each PSA to 
rank in the top 10 percent of the Nielsen ratings because this is the only 


oo 

The social media mentions metric can capture negative publicity about VHA or the VCL. 
For example, it could include Instagram or Facebook posts about unsatisfactory 
experiences with the VCL, such as long wait times. 
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meaningful target available that is accepted industry-wide.^"^ VHA officials 
stated that using any other targets would be arbitrary. For the remaining 
metrics, VHA officials told us they assess the outcomes of their campaign 
by comparing data from year to year, and examining any changes in the 
outcomes over time.^® However, VHA could set targets that capture the 
number of people who viewed or interacted with its outreach content, 
similar to its Nielsen target set for television viewership. Doing so would 
help VHA evaluate whether the campaign has been effective in raising 
awareness of VHA’s suicide prevention resources. Further, creating 
targets for these additional metrics need not be arbitrary, because VHA 
could use information about how its metrics performed in the past to 
develop reasonable and meaningful targets for future performance. VHA 
could also adjust the targets over time to reflect changes in its metrics or 
approach to the campaign, such as changes to its paid media budget 
each year. 

Federal internal control standards for monitoring require agencies to 
assess the quality of its performance by evaluating the results of 
activities. Agencies can then use these evaluations to determine the 

Apart from noting the lack of meaningful targets for their metrics, officials also noted that 
it is generally difficult to determine whether behavioral shifts have occurred in veterans as 
a result of VA’s outreach efforts and that they have struggled to study these types of 
outcomes. For example, in 2017, OMHSP proposed collaborating with VA’s Center of 
Excellence for Suicide Prevention, whose mission is in part to use research to inform the 
implementation of effective veteran suicide prevention strategies, to (1) examine the 
metrics for the suicide prevention media outreach campaign, and (2) obtain veterans 
perspectives on the campaign. However, officials said that this effort did not commence 
due to the lack of senior leadership during and after this time. Officials indicated they 
would like to continue with their evaluation efforts in the future. VA’s Center of Excellence 
has completed other studies that have examined the effects of VA’s suicide prevention 
campaign. For example, one study that examined associations between VA’s suicide 
prevention campaign and the use of crisis support services found that there were 
significant increases in call volume to the VCL to both targeted and broad resources 
during the campaign. See Elizabeth Karras, Naiji Lus, Guoxin Zuo, Xin M. Tu, Brady 
Stephens, John Draper, Caitlin Thompson, and Robert M. Bossarte, “Measuring 
Associations of the Department of Veterans Affairs’ Suicide Prevention Campaign on the 
Use of Crisis Support Services,” Suicide and Life-Threatening Behavior, 46 (4) August 
2016: 447-456. 

^^VHA officials shared that their general approach has been to improve their suicide 
prevention media outreach activities each year by increasing the number of materials 
issued and website visits because, officials said, this means more people are being 
reached. They said that the indirect result of reaching more people is that the number of 
VCL calls are increasing, meaning more people are aware of VA’s outreach activities. 
However, officials also recognized the number of visits to the website will likely reach a 
maximum after a certain point, and their goal might shift to maintaining the number of 
website visits rather than increasing it. 
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effectiveness of its programs or need for any corrective actions.^® Further, 
VA’s June 2018 National Strategy for Preventing Veteran Suicide also 
emphasizes the importance of the agency evaluating the effectiveness of 
its outreach.®^ The absence of established targets leaves VHA without a 
framework to effectively evaluate its campaign. Our prior work has shown 
that establishing targets allows agencies to track their progress toward 
specific goals. In particular, we have developed several key attributes of 
performance goals and measures including, when appropriate, the 
development of quantifiable, numerical targets for performance goals and 
measures.®® Such targets can facilitate future evaluations of whether 
overall goals and objectives were achieved by allowing for comparisons 
between projected performance and actual results. Further, establishing 
targets for its outreach metrics will enable VHA officials to determine 
whether outreach performed as expected and raised awareness of VHA 
resources such as the VCL, including identifying outreach efforts that 
worked particularly well and those that did not. In doing so, VHA officials 
will have the opportunity to make better informed decisions in their suicide 
prevention media outreach campaign to support VA’s overall goal of 
reducing veteran suicides. 

Conclusions 

VA has stated that preventing veteran suicide is its top clinical priority; yet 
VHA’s lack of leadership attention to its suicide prevention media 
outreach campaign in recent years has resulted in less outreach to 
veterans. While VHA identifies the campaign as its primary method of 
raising suicide prevention awareness, it has not established an effective 
oversight approach to ensure outreach continuity. This became 
particularly evident during a recent period of turnover and reorganization 
in the office responsible for the suicide prevention outreach campaign. 
Moving forward, VHA has an opportunity to improve its oversight to 
ensure that its outreach content reaches veterans and others in the 
community to raise awareness of VHA’s suicide prevention services, 
particularly as VHA begins working with a new contractor beginning in 
fiscal year 2019. 


®®GAO-14-704G and GAO/AIMD-00-21.3.1. 

®^VA, National Strategy for Preventing Veteran Suicide 2012-2018. 

®®GAO-03-143 and GAO-12-208G. 
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VHA is responsible for evaluating the effectiveness of its suicide 
prevention media outreach campaign in raising awareness about VHA 
services that are available to veterans who may be at risk for suicide. To 
do so, VHA collects and monitors data on campaign metrics to help 
gauge the effectiveness of its suicide prevention media outreach 
campaign in raising such awareness, but has not established targets for 
the majority of these metrics because officials reported that there are no 
meaningful, industry-wide targets for them. We disagree with VHA’s 
assertion that other targets would not be meaningful; VHA collects data 
on its metrics that it can use to develop reasonable and meaningful 
targets for future performance. In the absence of established targets, 
VHA cannot evaluate the effectiveness of the campaign, and make 
informed decisions about which activities should be continued to support 
VA’s overall goal of reducing veteran suicides. 


RSCOmmSndStionS for making the following two recommendations to VA: 

Executive Action i. The Under Secretary for Health should establish an approach for 

overseeing its suicide prevention media outreach efforts that includes 
clear delineation of roles and responsibilities for those in leadership 
and contract oversight roles, including during periods of staff turnover 
or program changes. (Recommendation 1) 

2. The Under Secretary for Health should require officials within the 
Office of Suicide Prevention and Mental Health to establish targets for 
the metrics the office uses to evaluate the effectiveness of its suicide 
prevention media outreach campaign. (Recommendation 2) 


Agency Comments 
and Our Evaluation 


We provided a draft of this report to VA for review and comment. In its 
written comments, summarized below and reprinted in Appendix I, VA 
concurred with our recommendations. VA described ongoing and planned 
actions and provided a timeline for addressing our recommendations. VA 
also provided technical comments, which we incorporated as appropriate. 

In response to our first recommendation, to establish an oversight 
approach that includes delineation of roles and responsibilities, VA 
acknowledged that organizational transitions and realignments within 
OMHSP contributed to unclear roles and responsibilities in 2017 and 
2018. VA said that OMHSP has made organizational improvements, 
including hiring a permanent Director for Suicide Prevention and 
establishing a new organizational structure. In its comments, VA 
requested closure of the first recommendation based on these actions. 
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However, to fully implement this recommendation, VA will need to provide 
evidence that it has established an oversight approach for the suicide 
prevention media outreach campaign. This would include providing 
information about the roles and responsibilities, as well as reporting 
requirements, for contract and leadership officials involved in the suicide 
prevention media outreach campaign under the new organizational 
structure and the new contract. VA will also need to demonstrate that it 
has a plan in place to ensure continued oversight of the suicide 
prevention media campaign in the event of staff turnover or program 
changes. 

In response to our second recommendation, to establish targets against 
which to evaluate suicide prevention metrics, VA said it has plans to work 
with communications experts to develop metrics, targets, and an 
evaluation strategy to improve its evaluation of its suicide prevention 
program efforts, including outreach. VA expects to complete these actions 
by April 2019. 


As agreed with your office, unless you publicly announce the contents of 
this report earlier, we plan no further distribution until 30 days from the 
report date. At that time, we will send copies to the appropriate 
congressional committees and the Secretary of Veterans Affairs. In 
addition, the report will be available at no charge on the GAO website at 
http://www.gao.gov. 

If you or your staff have any questions about this report, please contact 
me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. GAO staff who made key contributions to this 
report are listed in appendix I. 

Sincerely yours. 



Debra A. Draper 
Director, Health Care 
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Appendix I: Comments from the Department 
of Veterans Affairs 


THE SECRETARY OF VETERANS AFFAIRS 
WASHINGTON 

October 31, 2018 


Ms. Debra A. Draper 

Director 

Health Care 

U.S. Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Ms. Draper; 

The Department of Veterans Affairs (VA) has reviewed the Government 
Accountability Office (GAO) draft report: “VA HEALTH CARE: Improvements 
Needed in Suicide Prevention Media Outreach Oversight and Evaluation” (GAO- 
19-66). 

The enclosure contains technical comments and sets forth the actions to be 
taken to address the draft report recommendations. 

VA appreciates the opportunity to comment on your draft report. 

Sincerely, 

Robert L. Wilkie 

Enclosure 
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Appendix I: Comments from the Department of Veterans 
Affairs 


Enclosure 

Department of Veterans Affairs (VA) Comments to 
Government Accountability Office (GAO) Draft Report 

“VA HEALTH CARE: Improvements Needed in Suicide Prevention Media 
Outreach Campaign Oversight and Evaluation” 

(GAO-19-66) 

GAO Recommendation 1 : The Under Secretary for Health should establish an 
approach for overseeing its suicide prevention media outreach efforts that 
includes delineation of roles and responsibilities for those in leadership and 
contract oversight roles, including during periods of staff turnover or program 
changes. 

VA Comment : Concur. Suicide prevention is the Department of Veterans Affairs’ (VA) 
highest priority. Every death by suicide is a tragedy, and we will not relent in our efforts 
to connect Veterans in need with life-saving support. VA’s goal is to reduce suicide 
among Veterans. We can save lives. 

VA’s published National Strategy for Preventing Veteran Suicide provides the 
framework for identifying priorities, organizing efforts, and contributing to a national 
focus on Veteran suicide prevention over the next decade. The strategy aligns with the 
2012 National Strategy for Suicide Prevention, consists of 4 strategic directions, 14 
goals, 43 objectives, leverages the public health approach to suicide prevention, and 
focuses on the importance of collaboration and urgency. 

The four strategic directions are: 

1. Healthy and Empowered Veterans. Families, and Communities 

Work with community partners and policymakers to promote resources for Veterans 
and to promote mental health awareness and treatment. 

2. Clinical and Community Preventive Services 

Share resources, training opportunities, inten/entions, and lethal means safety 
resources with providers outside VA and with clergy, first responders, law 
enforcement officials, and community partners. 

3. Treatment. Recovery, and Support Services 

Work with local emergency departments and hospitals to promote Veteran suicide 
prevention resources and to ensure continuity of care for all Veterans admitted to an 
inpatient mental health unit. 

4. Sun/eillance. Research, and Evaluation 

Share the VA National Suicide Data Report and National Strategy for Preventing 
Veteran Suicide with community partners. 

VA has a comprehensive approach to suicide prevention. Here are just a few of our 
substantial efforts: 
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Appendix I: Comments from the Department of Veterans 
Affairs 


Enclosure 

Department of Veterans Affairs (VA) Comments to 
Government Accountability Office (GAO) Draft Report 
“VA HEALTH CARE: Improvements Needed in Suicide Prevention Media 
Outreach Campaign Oversight and Evaiuation” 

{GAO-19-66) 

1. Expanding the Veterans Crisis Line that connects Veterans in crisis and their 
families and friends with qualified, caring VA responders through a confidential toll- 
free hotline, online chat, or text. Veterans and their loved ones can call 
1-800-273-8255 and Press 1, chat online, or send a text message to 838255 to 
receive confidential support 24 hours a day, 7 days a week, 365 days a year. More 
information is available at https://www.veteranscrisisline.net/. 

2. Creating new cross-sector partnerships across various disciplines: faith 
communities, employers, schools, and health care organizations to name a few. 
Together we can reach more people. 

3. Implementing the Joint Action Plan for Supporting Veterans During Their Transition 
From Uniformed Service to Civilian Life 

4. Launching S.A.V.E. online suicide prevention training that educates yourself or your 
employees about how to recognize suicide risk factors and warning signs, 

5. Implementing the Mayor’s Challenge , which equips communities to build coalitions 
to prevent Veteran suicide. This partnership between VA and the Department of 
Health and Human Sen/ice Substance Abuse and Mental Health Sen/ices 
Administration (SAMHSA) helps local leaders in city governments work together to 
prevent suicide among Veterans. We have 27 Mayors now and we want more. 

6. #BeThere prevention initiative teaches members of the community how simple acts 
can help save a Veteran in crisis. More information is available at 
https://www.veteranscrisisline.net/BeThere.aspx. 

7. Make the Connection campaign is an online resource that connects Veterans, their 
family members and friends, and other supporters with information and solutions to 
issues affecting their lives. More information is available at 
https://maketheconnection. net/. 

8. Coaching into Care (1-888-823-7458) is a national telephone service of VA. 
Coaching into Care aims to educate, support, and empower family members and 
friends who are seeking care or services for a Veteran. More information is 
available at https://www.mirecc.va.gov/coaching/. 

9. Recovery Engagement and Coordination for Health - Veterans Enhanced Treatment 
Program (REACH VET) is an innovative program that applies principles of predictive 
medicine to prevent suicide. 
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Appendix I: Comments from the Department of Veterans 
Affairs 


Enclosure 


Department of Veterans Affairs (VA) Comments to 
Government Accountability Office (GAO) Draft Report 

“VA HEALTH CARE; Improvements Needed in Suicide Prevention Media 
Outreach Campaign Oversight and Evaiuation” 

(GAO-19-66) 

1 0. About Face videos feature the reai stories of Veterans who have experienced post- 
traumatic stress disorder with additional perspectives from their family members and 
VA clinicians. https://www.ptsd.va.gov/apps/AboutFace/. 

11 . How to Talk to a Child About a Suicide Attempt in Your Family : If there has been a 
recent suicide attempt in your family, this may be one of the toughest experiences 
you and your children may ever face. It is important to take care of yourself, so that 
you are better able to care for your child. 

12. Paid media outreach such as advertisements on digital and social media. 

VA’s suicide prevention efforts are successful: 

• Over 16,000 American citizens have downloaded VA’s S.A.V.E. training to prevent , 
suicides. Annual S.A.V.E. training is mandatory for VA employees, and in 
September 2018 alone over 52,000 VA employees completed it. 

• The number of Veterans receiving mental health care in VA doubled over the past 
10 years, now we are helping over 1.7 million Veterans with their mental health 
concerns. 

• VA repeatedly meets our goals for being in the top 10 percent Nielsen ranking for 
our suicide prevention public service announcements. 

• The Veterans Crisis Line serves about 2000 callers a day. Over the past 10 years, 
the Veterans Crisis Line has answered over 3 million calls, more than 363,000 online 
chats, and more than 83,000 text messages. 

• Last year, VA’s suicide prevention coordinators did 18,836 community outreach 
events and reached approximately 1,932,902 people. 


Over 102,000 people visited our #BeThere campaign page last year. 

Twenty-seven mayors are taking the challenge to reduce Veteran suicide in their 
cities. VA wants every city to take action to prevent suicide. 

During Suicide Prevention Month (September 2018) VA saw a lot of success with 
#BeThere messaging: 

o 50+ million #BeThere Twitter Chat impressions 
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Appendix I: Comments from the Department of Veterans 
Affairs 


Enclosure 

Department of Veterans Affairs (VA) Comments to 
Government Accountability Office (GAO) Draft Report 

“VA HEALTH CARE: Improvements Needed in Suicide Prevention Media 
Outreach Campaign Oversight and Evaiuation” 

(GAO-19-66) 

o 18 million+ social media reach (over past 7 days) 

o 2.6 million Thunderclap social media reach 

o 1,800+ registrants for Action Alliance's Messaging Webinar 

o 100+ private and public sector partners 

In 2018, VA partnered with Johnson & Johnson on a broadly promoted public service 
announcement featuring Tom Hanks speaking that GAO did not include in the draft 
report. YouTube alone had over 30,000 views. VA showcased it as part of the 
#BeThere campaign. This outreach effort was supported through a non-monetary 
Memorandum of Agreement with Johnson & Johnson, at no cost to the Veterans Health 
Administration (VHA). 

Regarding Paid Media spending, VHA estimates that the total amount spent in fiscal 
year (FY) 2018 will be about $1.5M, This will include money spent in September 2018 
for Suicide Prevention Month paid media. VHA will not have final invoices for an exact 
total until sometime after October 2018. As indicated above, paid media is only one 
component of VHA’s comprehensive suicide prevention effort, and it is a small portion of 
VHA’s annual expenditures for suicide prevention. 

VHA acknowledges that organizational transitions and realignments within Office of 
Mental Health and Suicide Prevention (OMHSP) contributed to unclear roles and 
responsibilities from mid-2017 to mid-2018. OMHSP has made significant 
organizational structure improvements to clearly delineate roles and responsibilities for 
those in leadership and contract oversight roles since that time. Organizational 
improvements within the Executive Leadership team and the Suicide Prevention team 
also ensure continuity during periods of staff turnover and program changes. OMHSP 
has hired a permanent Director for Suicide Prevention, and established and received 
approval for a permanent organizational structure which creates a clear chain of 
command within OMHSP and to VHA. With this new robust organizational structure in 
place, the Director for Suicide Prevention oversees communications and outreach 
activities in a more organized and deliberate manner. Similarly, contractual tasks 
related to these outreach activities fall within the Deputy Director for Partnerships’ 
portfolio. 

VHA has completed work on this recommendation and requests GAO close this 
recommendation as implemented. In support of our closure request, VHA has 
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submitted the final, signed OMHSP organizational chart to GAO. Completion Date; 

July 2018 

Recommendation 2 : The Under Secretary for Health should require officials 
within the Office of Suicide Prevention and Mentai Health to establish targets for 
the metrics the office uses to evaiuate the effectiveness of its suicide prevention 
media outreach campaign. 

VA Comment : Concur. GAO recognized that OMHSP has a robust monitoring 
process in place to track both reach and engagement of our suicide prevention outreach 
campaigns. Beyond quantitative data, OMHSP also tracks qualitative measures of 
success such as establishing partnerships with prominent organizations or securing 
promotion of our resources from high-profile public figures. 

VHA has looked at broad indicators of impact for its suicide prevention outreach efforts 
for some time. VHA’s current Agency Priority Goal is to provide core recommended 
interventions to Veterans targeted through its predictive risk modeling algorithms and to 
reduce Veteran suicide through a “Mayor’s Challenge” partnership program. The 
growth in demand for services at the Veterans Crisis Line (VCL) has grown year over 
year and has been supported through ongoing outreach. In particular over FY 2018, the 
demand for chat and text services at VCL increased by over 25 percent and nearly 30 
percent, respectively. Local outreach activities have also been part of national strategy 
including annual Community Mental Health Summits at VHA facilities. In 2018 there 
were 149 Summits with over 13,500 participants, and 83 percent of participants 
reported gaining a better understanding of suicide prevention resources and services 
through VHA and in their community. 

VHA accepts GAO's recommendation to do more. OMHSP has begun to leverage VA 
expertise in health communication and evaluation to determine additional measures of 
effectiveness. These measures will be benchmarked against similar national program 
metrics. This effort is part of a larger initiative that is underway to determine program 
measures of effectiveness for the entire suicide prevention program. After receiving 
results from communications experts, OMHSP will finalize metrics, targets and the 
subsequent evaluation strategy. Target Completion Date: April 2019. 


5 


Page 30 


GAO-19-66 VA Suicide Prevention Media Outreach 





Appendix II: GAO Contact and Staff 
Acknowledgments 



GAO Contact 

Debra A. Draper, (202) 512-7114 or draperd@gao.gov 

Staff 

Acknowledgments 

In addition to the contact named above, Marcia A. Mann (Assistant 
Director), Kaitlin McConnell (Analyst-in-Charge), Kaitlin Asaly, and Jane 
Eyre made key contributions to this report. Also contributing were Jennie 
Apter, Emily Bippus, Valerie Caracelli, Lisa Gardner, Jacquelyn Hamilton, 
Teague Lyons, Vikki Porter, and Eden Savino. 


( 102479 ) 


Page 31 


GAO-19-66 VA Suicide Prevention Media Outreach 








GAO’S Mission 

The Government Accountability Office, the audit, evaluation, and investigative 
arm of Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability of the 
federal government for the American people. GAO examines the use of public 
funds; evaluates federal programs and policies; and provides analyses, 
recommendations, and other assistance to help Congress make informed 
oversight, policy, and funding decisions. GAO's commitment to good government 
is reflected in its core values of accountability, integrity, and reliability. 

Obtaining Copies of 
GAO Reports and 
Testimony 

The fastest and easiest way to obtain copies of GAO documents at no cost is 
through GAO’s website (https://www.gao.gov). Each weekday afternoon, GAO 
posts on its website newly released reports, testimony, and correspondence. To 
have GAO e-mail you a list of newly posted products, go to https://www.gao.gov 
and select “E-mail Updates.” 

Order by Phone 

The price of each GAO publication reflects GAO’s actual cost of production and 
distribution and depends on the number of pages in the publication and whether 
the publication is printed in color or black and white. Pricing and ordering 
information is posted on GAO’s website, https://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, MasterCard, 

Visa, check, or money order. Call for additional information. 

Connect with GAO 

Connect with GAO on Facebook, Flickr, Twitter, and YouTube. 

Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts. 

Visit GAO on the web at https://www.gao.gov. 

To Report Fraud, 

Waste, and Abuse in 
Federal Programs 

Contact: 

Website: https://www.gao.gov/fraudnet/fraudnet.htm 

Automated answering system: (800) 424-5454 or (202) 512-7700 

Congressional 

Relations 

Orice Williams Brown, Managing Director, WilliamsO@gao.gov, (202) 512-4400, 
U.S. Government Accountability Office, 441 G Street NW, Room 7125, 
Washington, DC 20548 

Public Affairs 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Strategic Planning and 
External Liaison 

James-Christian Blockwood, Managing Director, spel@gao.gov, (202) 512-4707 
U.S. Government Accountability Office, 441 G Street NW, Room 7814, 
Washington, DC 20548 


Please Print on Recycled Paper. 



















